Skip to content

kernel: clarify and assert interrupts aren't disabled upon context switch #93440

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

mathieuchopstm
Copy link
Contributor

The do_swap() routine used when CONFIG_USE_SWITCH=y asserts that caller thread does not hold any spinlock when CONFIG_SPIN_VALIDATE is enabled. However, there is no similar check in place when CONFIG_USE_SWITCH=n.

Copy this assertion in the USE_SWITCH=n implementation of z_swap_irqlock().

The do_swap() routine used when CONFIG_USE_SWITCH=y asserts that caller
thread does not hold any spinlock when CONFIG_SPIN_VALIDATE is enabled.
However, there is no similar check in place when CONFIG_USE_SWITCH=n.

Copy this assertion in the USE_SWITCH=n implementation of z_swap_irqlock().

Signed-off-by: Mathieu Choplain <mathieu.choplain@st.com>
andyross
andyross previously approved these changes Jul 21, 2025
Copy link
Contributor

@andyross andyross left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For clarity: this is useful for architectures like arm32 that have an arch_swap() and miss the check elsewhere. The error being detected is always wrong.

krish2718
krish2718 previously approved these changes Jul 21, 2025
Copy link
Contributor

@krish2718 krish2718 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this caught the bug in my case where a thread holding the spinlock tries to sleep (mutex).

peter-mitsis
peter-mitsis previously approved these changes Jul 21, 2025
TaiJuWu
TaiJuWu previously approved these changes Jul 22, 2025
@mathieuchopstm
Copy link
Contributor Author

For the record, this patch breaks part of the irq_lock() specification when CONFIG_SPIN_VALIDATE=y since there's no way to distinguish between k_spin_lock() and irq_lock():

zephyr/include/zephyr/irq.h

Lines 242 to 251 in 1f69b91

* @note
* This routine can be called by ISRs or by threads. If it is called by a
* thread, the interrupt lock is thread-specific; this means that interrupts
* remain disabled only while the thread is running. If the thread performs an
* operation that allows another thread to run (for example, giving a semaphore
* or sleeping for N milliseconds), the interrupt lock no longer applies and
* interrupts may be re-enabled while other processing occurs. When the thread
* once again becomes the current thread, the kernel re-establishes its
* interrupt lock; this ensures the thread won't be interrupted until it has
* explicitly released the interrupt lock it established.

Using QEMU, I verified that the existing assertion (when CONFIG_USE_SWITCH=y, from bd07756) does panic if the irq_lock() is held upon context switch:

diff --git a/samples/hello_world/src/main.c b/samples/hello_world/src/main.c
index c550ab461cb..be87d15501c 100644
--- a/samples/hello_world/src/main.c
+++ b/samples/hello_world/src/main.c
@@ -5,10 +5,13 @@
  */
 
 #include <stdio.h>
+#include <zephyr/kernel.h>
 
 int main(void)
 {
+       irq_lock();
        printf("Hello World! %s\n", CONFIG_BOARD_TARGET);
+       k_msleep(1000);
 
        return 0;
 }
$ # with upstream
$ west build -b qemu_x86_64 samples/hello_world/ -DCONFIG_USE_SWITCH=y -DCONFIG_ASSERT=y -DCONFIG_SPIN_VALIDATE=y -p
$ west build -t run
-- west build: running target run
[3/4] To exit from QEMU enter: 'CTRL+a, x'[QEMU] CPU: qemu64,+x2apic
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.x2apic [bit 21]
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.x2apic [bit 21]
SeaBIOS (version zephyr-v1.0.0-0-g31d4e0e-dirty-20200714_234759-fv-az50-zephyr)
Booting from ROM..
*** Booting Zephyr OS build v4.2.0-334-g124fb897b490 ***
Hello World! qemu_x86_64/atom
ASSERTION FAIL [arch_irq_unlocked(key) || z_smp_current_get()->base.thread_state & (((1UL << (0))) | ((1UL << (3))))] @ WEST_TOPDIR/zephyr/kernel/include/kswap.h:98
        Context switching while holding lock!
RAX: 0x0000000000000004 RBX: 0x000000000011fd90 RCX: 0x0000000000000001 RDX: 0x0000000000000000
RSI: 0x0000000000000062 RDI: 0x00000000001090d2 RBP: 0x000000000012bd70 RSP: 0x000000000012bd48
 R8: 0x0000000000000000  R9: 0x0000000000000000 R10: 0x0000000000000002 R11: 0x0000000000000000
R12: 0x000000000010b500 R13: 0x0000000000000000 R14: 0x0000000000000000 R15: 0x000000000012be48
RSP: 0x000000000012bd48 RFLAGS: 0x0000000000000002 CS: 0x0018 CR3: 0x0000000000136000
RIP: 0x000000000010046f
call trace:
     0: 0x00000000001051d5
     1: 0x000000000010524d
     2: 0x0000000000100025
     3: 0x0000000000102f2d
     4: 0x000000000010045a

On a related note, the k_spin_lock() documentation does not mention whether or not holding a k_spinlock is allowed upon reschedule:

/**
* @brief Lock a spinlock
*
* This routine locks the specified spinlock, returning a key handle
* representing interrupt state needed at unlock time. Upon
* returning, the calling thread is guaranteed not to be suspended or
* interrupted on its current CPU until it calls k_spin_unlock(). The
* implementation guarantees mutual exclusion: exactly one thread on
* one CPU will return from k_spin_lock() at a time. Other CPUs
* trying to acquire a lock already held by another CPU will enter an
* implementation-defined busy loop ("spinning") until the lock is
* released.
*
* Separate spin locks may be nested. It is legal to lock an
* (unlocked) spin lock while holding a different lock. Spin locks
* are not recursive, however: an attempt to acquire a spin lock that
* the CPU already holds will deadlock.
*
* In circumstances where only one CPU exists, the behavior of
* k_spin_lock() remains as specified above, though obviously no
* spinning will take place. Implementations may be free to optimize
* in uniprocessor contexts such that the locking reduces to an
* interrupt mask operation.
*
* @param l A pointer to the spinlock to lock
* @return A key value that must be passed to k_spin_unlock() when the
* lock is released.
*/

The options I see are:

  1. don't merge this patch ( 😢 )
  2. merge patch as-is
  • Holders of irq_lock on context switch will inexplicably panic if CONFIG_SPIN_VALIDATE=y
  • ...and it should be remarked that CONFIG_SPIN_VALIDATE usually defaults to y if CONFIG_ASSERT=y
  1. update documentation of irq_lock() to indicate that holding irq_lock on context switch will panic when CONFIG_SPIN_VALIDATE=y
  2. update documentation of irq_lock() to no longer allow holding irq_lock on context switch (breaking change?)

@andyross
Copy link
Contributor

My vote would just be to remove that section of the docs and rip the bandaid off with the warning (which can always be disabled via kconfig anyway). There are precisely zero circumstances where breaking a critical section (!) lock with a context switch (!!) cannot lead to tears.

Code that does that is not only breaking its own carefully curated lock in an immensely clever way that it clearly thought carefully about and knows has no unexpected interactions. Because of the recursive behavior of irq_lock() it's also breaking the locks taken by all the callers up the stack, who had no idea that calling into this obnoxiously clever subsystem was a synchronization trap.

It just can't work. We never should have documented it (and I didn't even know we had!).

@mathieuchopstm
Copy link
Contributor Author

mathieuchopstm commented Jul 23, 2025

What do you think of replacing the existing note:

zephyr/include/zephyr/irq.h

Lines 242 to 251 in 47b07e5

* @note
* This routine can be called by ISRs or by threads. If it is called by a
* thread, the interrupt lock is thread-specific; this means that interrupts
* remain disabled only while the thread is running. If the thread performs an
* operation that allows another thread to run (for example, giving a semaphore
* or sleeping for N milliseconds), the interrupt lock no longer applies and
* interrupts may be re-enabled while other processing occurs. When the thread
* once again becomes the current thread, the kernel re-establishes its
* interrupt lock; this ensures the thread won't be interrupted until it has
* explicitly released the interrupt lock it established.

with the following?

 * @note
 * This routine can be called by ISRs or by threads.
 * Only isr-ok functions may be called while holding the interrupt lock.
 * It is a fatal error to hold the interrupt lock upon return from ISR.

These statements sound correct from my understanding of the kernel:

  • isr-ok functions are the only ones guaranteed to not context switch
  • Kernel might reschedule upon return-from-ISR

We could add a more direct It is a fatal error to hold the interrupt lock upon context switch but I feel like these statements are equivalent, while being easier to understand from people not necessarily familiar with kernel terminology.

If this is fine for you, I'll add a new commit that edits irq.h.

@krish2718
Copy link
Contributor

It is a fatal error to hold the interrupt lock upon context switch

How about the infamous Linux wording BUG: Scheduling while atomic?

@mathieuchopstm
Copy link
Contributor Author

It is a fatal error to hold the interrupt lock upon context switch

How about the infamous Linux wording BUG: Scheduling while atomic?

We already have a runtime message 🙂

__ASSERT(arch_irq_unlocked(key) ||
_current->base.thread_state & (_THREAD_DUMMY | _THREAD_DEAD),
"Context switching while holding lock!");

What needs updating is the documentation for irq_lock() as it isn't consistent with the rest of the kernel.

@bjarki-andreasen
Copy link
Contributor

I think we need the explicit wording of "It is a fatal error to hold the interrupt lock upon context switch", since a thread can force a reschedule/context switch at any time by calling k_yield() for example.

@mathieuchopstm
Copy link
Contributor Author

I think we need the explicit wording of "It is a fatal error to hold the interrupt lock upon context switch", since a thread can force a reschedule/context switch at any time by calling k_yield() for example.

I don't think k_yield() is isr-ok though?

(Really, my Only isr-ok functions may be called while holding the interrupt lock is a non-kernel-wording way of saying It is a fatal error to hold the interrupt lock upon context switch - I don't mind adding it anyways if you feel like it's better though)

@bjarki-andreasen
Copy link
Contributor

bjarki-andreasen commented Jul 23, 2025

I think we need the explicit wording of "It is a fatal error to hold the interrupt lock upon context switch", since a thread can force a reschedule/context switch at any time by calling k_yield() for example.

I don't think k_yield() is isr-ok though?

(Really, my Only isr-ok functions may be called while holding the interrupt lock is a non-kernel-wording way of saying It is a fatal error to hold the interrupt lock upon context switch - I don't mind adding it anyways if you feel like it's better though)

I disagree with the way the note is worded, since when I'm reading it, I'm reading the three lines as a strangely phrased paragraph, and in my mind auto-correcting it to make it make sense. Could you split the three notes into three notes, or phrase it as a paragraph?

 * @note This routine can be called by ISRs or by threads.
 * @note Only isr-ok functions may be called while holding the interrupt lock.
 * @note It is a fatal error to hold the interrupt lock upon return from ISR.

or

* @note This function can be called by ISRs and threads. After calling this
* routine, the caller is "holding the interrupt lock". While holding the interrupt lock,
* only isr-ok functions may be called, as these don't result in context switches.
* While holding the interrupt lock, a context switch, which could result from calling
* a non isr-ok function, or returning from an ISR, is a fatal error.

@andyross
Copy link
Contributor

Bikeshed aside, I think adding a warning in the docs is fine, but IMHO it should be as short and clear as possible. "Context switching while holding the irq_lock is illegal." is fine, etc... The user will be alerted by the assert if they trip over it. (Also nitpick: it's not a "fatal error" as nothing actually fails or panics in an obvious way, which is why this is a booby trap.)

Upon context switch, the virtual "interrupt lock" acquired by irq_lock()
must not be "held". However, the current documentation for irq_lock() says
that it is perfectly valid to hold it (!), and that a suspended thread will
hold the "interrupt lock" upon being scheduled again (!!).

Update the documentation to remove the outdated section and indicate that
context switching while holding the interrupt lock is not allowed.

Signed-off-by: Mathieu Choplain <mathieu.choplain@st.com>
Threads must not attempt to context switch if they are holding a spinlock.

Add this information to the documentation for k_spin_lock().

Signed-off-by: Mathieu Choplain <mathieu.choplain@st.com>
@mathieuchopstm mathieuchopstm dismissed stale reviews from TaiJuWu and bjarki-andreasen via a173881 July 23, 2025 14:23
@mathieuchopstm mathieuchopstm dismissed stale reviews from peter-mitsis, krish2718, and andyross via a173881 July 23, 2025 14:23
@mathieuchopstm
Copy link
Contributor Author

Modified the irq_lock() (and k_spin_lock() while at it) documentation with short comments.

Copy link

Copy link
Contributor

@andyross andyross left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks clear to me.

@mathieuchopstm mathieuchopstm changed the title kernel: assert no spinlock is held on swap when !USE_SWITCH kernel: clarify and assert interrupts aren't disabled upon context switch Jul 23, 2025
Copy link
Contributor

@bjarki-andreasen bjarki-andreasen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems perfect

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants